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I 

INTRODUCTION 

This paper may be viewed from three major perspectives. It reports 
the identification of an educational production function by a method pre- 
viously unused for that purpose, and in stricter conformity with economic 
theory than most earlier educational production function stujdies. It also 
deals with a relationship which has received too little attention in mo*^t 
studies of school performance or student achievement: namely, the relation 

^^^^ allocation of time to student achievement . In addition, this paper 
may be viewed as a report on the use of a mastery learning strategy. 

The raoti.>/ation for each of these three components was described in a 
paper presented to this Association two years ago.^ In the interest of 
brevity, I shall consider that earlier paper to be the introduction to the 
present one. Readers who desire a fuller explanation of the background 
and rationale of my work are urged to consult that earlier paper. 

In brief, my argument is that most previous educational production 
function studies suffer from a number of shortcomings: 1) improper speci- 
fication of outputs; 2) inattention to the distribution of outputs; 3) fail- 
ure to use longitudinal data on changes in outputs (indeed, the use of cros3- 
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What follows is a preliminary and Bunanary description of a project to be re- 
ported in detail in The Identification of an Educational Production Function 
by Experimental Means > Ph.D. dissertation in progress. The Department of Edu- 
cation. University of Chicago. 
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"Inputs and Outputs in the Educational Process/' a paper by this author pre- 
sented at a symposium on "What Can Economics Contribute to Education?/* AERA, 
New York, February 4, 1971. Available from the author. 
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sectional rather than longitudinal data on "value-added" means that many 
so-called "production-function" studies do not qualify on this ground 
alone); 4) improper specification of inputs; 5) the assumption that an 
average of school inputs is applied to each student; 6) the use of school, 
or even district-wide, averages rather than data for individual students; 

7) the assumption that reported stocks of inputs are fully utilized, 

8) the assumption that inputs are utilized efficiently; and 9) aggregation 
over diverse instructional technologies. 

In view of these many shortcomings, it is not surprising that educa- 
tional "production-function" studies have not led to more powerful or more 
useful results. Take, for example, the assumptior. that the average amount 
of each school input is applied to each studenc. To the extent that thi. 
represents actual educational practice, it is an Important reaso: to ex- 
pect background variables to account for more of the variance in achieve- 
ment than school variables, except when schools show large variance in in- 
put means. And to the extent it doe.- not reflect actual practice, it, o! 
course, weakens the models, particularly with respect to the application 
of results to improve withln-school allocations. It must be kept in mind 
that production (or, education) is a process which takes place over time; 
it is not an instantaneous event. Thus if we are to know how much of a 
particular resource is used by students, we must have observations on the 
allocations of student time with those resources. 

In a very real sense, it is the student, rather than the school, who 
"produces" his own gains in achievement. Each student has his own array 
of talents and skills which he combines with available schooJ resources 
in order to "learn," i.e.. to produce increments in his measured achieve- 



ment. Thus again there is a need for data on the resources of individual 
students-their general ability and their prior achievement in specific 
areas~and on the time allocated by students to learning tasks. 

Finally, we must know the distributions of inputs and outputs, and 
the relation between them. With such knowledge, we can answer the ques- 
tion-Given a particular time allocation of school and student resources, 
what outputs can be expected?" Or, to pose the question asked in this re- 
search. "Given particular distributions of desired outputs, what time al- 
locations of school and student resources are required co produce them?" 

The information required for rigorous educational production studies 
cannot be easily obtained by observation of natural school environments or 
from conventional school records. Hence, I advocate an experimental ap- 
proach designed specifically to elicit such production information. It 
would undoubtedly be wasteful to attempt to investigate the production re- 
lationships in all educational situations by experimental means. But study 
of some key educational processes, such as early reading or arithmetic 
mastery, could yield information of importance. A sufficient nurier of 
such experimental studies might provide a basis not only for improving the 
efficiency of educational production, but for rationalizing the distribu- 
tion of achievement, especially achievement of the basic learning skills 
upon which later learning both in school and on the job are presuaably 
founded. Such experiments must Include: 

1) Clearly defined outputs and inputs. 
-) Specification of the technology, and 
3) Assurance of technical efficiency. 
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Condition one, with the present state of educational measurement, will 
lf.mit choice of outputs to a fairly narrow range of cognitive skills. This 
limitation, however, is a practical, not a conceptual one. Condition two 
must be understood in its broad sense, in which technology is not merely 
hardware of various kinds, but organizational and instructional procedures 
as well. Condition three is met largely through exercising care in the 
conduct of experiments, and is one of the reasons for preferring experi- 
mental to naturalistic settings. The present study attempts to meet all of 
these requirements. 

J\\i,s Study in Brief 

The general approach was to give three brief programmed instruction 
lessons on "matrix arithmetic" (matrix algebra) to about, 100 eighth grade 
public school students in two racially and socio-economically homogeneous 
schools. In addition to each student's pretest score on the specific sub- 
ject matter, his or her prior achievement scores on a standard test in 
reading and arithmetic were obtained from school records. Students were 
randomly assigned to one of nine possible treatment sequences; each treat- 
ment consisting of a required score on lesson tests. A record was kept by 
the student of time spent on each part of the sequence. The resulting data 
was subjected to regression analysis using a model assuming multiplicative 
interaction among the independent variables, and using elapsed time as de- 
pendent variable. 

Two basic elements of the technology employed were programmed in- 
struction format and a "mastery learning" strategy. The reader is assumed 



to be familiar with programmed instruction. The essential features of 
mastery learning''' are.* 

1) Stated behavioral objectives. These are predetermined things 
students ar«- intended to learn, that is, learning tasks which are to be 
mastered . 

2) Formative tests of each student's relevant prior achievement, 
before instruction begins, and of his progress toward the current objec- 
tives. Such tests are for informational purposes, not for assigning 
grades. 

3) Diagnosis of the deficiencies of each student with respect to 
the objective behaviors, based on the formative tests. 

4) Instructional sessions, which may include group and individual 
instruction. 

5) Individual (sometimes paired or group) review and remedial work, 
based on the diagnoses. 

6) Criterion performance levels. These are defined as proportions 
correct on tests of the objective behaviors. 

By using the mastery learning approach, it was possible to arrange and 
define the conditions of learning so that most students mastered specified 
skills at given levels of performance, a model which closely parallels the 
econpmic one of a firm which can prod ace a uniform product at vario is level 
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See James H. Block (ed.), Mastery Learning. Theory and Practice . (New 
York: Holt, Rinehart, and Winston, Inc., 1971. 
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Of output. ;n prcdactlon terms, tbo model employed here could be specified 
in part as follows: 

1) The uniform kind of output produced is mastery hy all (or nearly 
all) students of the defined skills (objectives) of a particular course or 
subject area. 

2) The level of output is the criterion performance level, as defined 
above. 

3) The t echnology of produ tion is programmed instruction, mastery 
learning, and other conditions and procedures detailed below. 

The next section of this paper describes the administrative and data- 
collection procedures of the experiment. Selected results will be pre- 
sented in section III. The results will be discussed and the paper con- 
eluded in Section IV. 



PROCEDURES 



This section describes the subjects and materials auployed in this 
-poriment. and summarizes the experimental and data-collection procedure 
l-hus it Ls in large part a specification of the relevant technology. 

Teaching MateMals 

Certain general considerations guided the search for " (.eachJ ng ma- 
terials appropriate for this study. First, a subject-matter was sought 
which would permit the clear, or relatively clear, statement of learning 
outcomes ("outpuf) in measurement tern.. This consideration made math.- 
-tics a natural candidate for use here. Second, a programmed format was 
desired, in order to render the presentation of nutcrial to students as 
Uniterm as possible. The object was to reduce teacher influences upon 
-student learning, as no characterization of "key" teacher attributes was 
to be attempted here. Third, the :„aterials should lend themselves to the 
manipulation of student achievement levels, a necessary condition fu. nhi.s 
experiment but one foreign to standard curriculum materials. 

These considerations were largely satisfied by the matrix arithmetic 
lessons developed by James H. Block for his dissertation research.^ lilock' 
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general approach, described in detail In his thesis, was to adapt the In- 
troductory lessons of a matrix algebra textbook to eightli-grade level. The 
principal problem was non to make the material easier to learn, but rather, 
more difficult! The original textbook, being In programmed format, was 
designed to provide "errorless" learning, through the provision of fre- 
quent practice and repetition in the text. The efficacy of the unmodified 
text is shown in Block's report tiiat, in a sample of "above average" eiv^th- 
graders, student performance on unit tests averaged 80 percent correct.^ ' 
By eliminating redundancies and making other changes, Block obtained a 
set of three lessons sech that "[t]he mean performance of students using 

the modified textbook, wherein both practice and repetition were minimal, 

2 

was only 50 percent." 

For each lesson, Block constructed a twenty-item test of lesson con- 
tent. Two additional equivalent verisions of each lesson test were then 
generated according to procedures described by Bormutli. Item-specific 
review and practice material was then constructed, so that a person mis- 
sing any given test item could be directed to review content specific to 
that item. Thus the complete array of available curricular material for 
each lesson consisted of: 
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Ibid., p. 48. 
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Ibid ., p. 50 
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John R. Bormuth, On the Theory of Achievement Test Item s, (Chicaog: Uni- 
versity of Chicago Press, 1970). ~ 



1) An Initial "text" or lesson; 

2) A fonnative test; 

3) Review material; 

A) Two additional versions of th.- formative test. 

The reader should now see how these materials allow manipulation of 
student achlevoment. that is. student mastery of lesson content as measured 
by scores on lesson test. The typical student using the modified text, can 
be expected to score perhaps 50 percent on the initial formative test of 
Twenty items. It is assumed that the student has already .mastered tUe con- 
tent ol the ten items he answered correctly. Suppose we desire the student 
to improve his mactery to 80 percent, or 16 Items correct. The student can 
be given review material specific to six missed items. He is not assigned 
more than .ix items to review since We do not (for purposes of the experi- 
ment) desire him to achieve more than 80 percent mastery. After review, 
he is tested again. If he still falls short of mastery, say by one item, 
he is given new review materials and subsequently another test. This se- 
quence was not continued indefinitely in this experiment; results by the 
third formative test were accepted as final for the particular lesson. How 
well this system was able to produce mastery at target levels will be re- 
ported in Section III. 

Subj ects 

An attempt „as made to locate a population of about IQO el^hth-graders 
of one race and with relatively homogeneous socio-economic characteristics^ 
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poses of comparison. Unfortunatplv ti^^ ^ a **'-«-«5i^ounas tor pur- 

experiment possible, ""'^^^^^^l^' «»°ney limitations made only one 
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A schoo] district organized on a "neighborhood school" basis in a resi- 
dential suburb west cf Chicago rooperated, but it was not possible to ar- 
range the study In just one school (due to the small size of the schools). 
Nor could the two schools in the 'most -.imilar" neighborhoods be included 
in the study. Thus, two schools in somewhat different parts of the com- 
munity were obtained. The resulting divergence in student socio-economic 
backgrounds is illustrated in Table 1, cotiparing student family size and 
parents' level of education. 

Table 1 

MEANS OF STUDENT FAMILY CHARACTERISTICS, BY SCHOOL 





(1> <2) (3) (A) 
No. Persons No. of Father's Mother's 

in Children Educ. Zduc. 
Household Under 18 (Years) (fparn) 


School in 


5.6 


3.1 


15.6 


IA.7 


School if 2 


5.7 


3.2 


13.1 


12.8 


T Value* 


.271 


.284 


A. 27** 


3.58** 



* T Value for difference of means test 
** Significant at better than .01 level (two tailed) 

Students In the two schools come from families which do not differ 
importantly in total size (Table 1, Column 1), or in the number of chil- 
dren under eighteen years eld (Column 2). The educational attainment cf 
fathers and mothers, however, does differ "significantly" between the 
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school (Col„^ 3 4,. I.e deuetabu effects of ..ese between .cbool 
dl/f.re„ces i„ parents' level of education .111 be diacusaed in tbe follow- 
ing section, 

A total of 135 students, comprising all the eighth-graders in the two 
schools, participated in the experiment, but due t. absences or incomplete 
records, only 110 students are Included In the analysis. 

All subjects are Caucasian. 

School A dministered Standard Tpsrs 

All subjects (for which co,^lete records ere available) took tbe Stan- 
ford Achievement Test. For. H, Advanced Battery, during October 1,70. Each 
=tudenfs raw scores on the Paragraph Meaning and Arithmetic Concepts sub- 
sections were obtained fro. school records. The SAT tests are designed with 
■•generous" ti.e-li.its. in order to be "fundamentally power tests and not 
speed tests. ""^ 



Experimental Desig n 

The teaching t^terials co.„rised three sequential lessons in ^trix 
arithmetic. The objeot was to have each student learn each lesson, but 
no. With the same level of mastery for all students. Thus, by a fundamen- 
tally arbitrary decision, three performance, or mastery, levels were selec- 
t-i as targets for each lesson. These were the 65.80, and 95 percent levels. 
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SAT Directions for Administering, p. 3. 
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which correspond to 13, 16, and 19 items correct on a 20-item test. The 
students of each school were separately ranked according to SAT Arithmetic 
Concept raw score, then taken sequentially in runs of three and randomly 
assigned to one of three groups. The groups were then randomly designated 
as 65, 80, and 95 percent raiistery groups. These assignments were recorded 
as each student's target joastery level for lessons one and two / The 
randomization was then repeated to assign students to target performance 
levels for lesson three. As a result of this procedure, some students 
(about 1/3) received the same performance level assignment lor all three 
lessons, while some received sequences which raised or lowered the target 
performance level after the second lesson. 

This random reassignment of performance level permits the investiga- 
tion of the effect of performance level sequence on learning. 

The experimental design may be illustrated as a nine-cell matrix, as 
in Table 2, The number in each cell indicate?: the number of students with 
the corresponding assignments," (Ignore the parenthetical T^ for the pre- 
sent,) The main diagonal of the matrix represents uniform performance level 
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Block found that, on the average, students took c.bout as long to complete 
both lessons one and two as to complete lesson three alone. Thus it was de- 
cided here to assign the same target mastery level to lessons one and two for 
each student. Some simplification of this kind was required by the limits of 
the sample size, 

2 

N«107, Complete data Is available for 110 subjects, but three students 
followed incorrect sequences during the experiment due to experimenter error. 
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assignments for all three lessons. Cells above the diagonal are assign- 
ments in which the performance level increases from the second to the 
third lesson; in cells below the diagonal, the sequence decreases. 



Table 2 

PERFORMANCE LEVEL ASSIGNMENT MATRIX 



Lesson Three 
Performance Level Assignment 



9 

> <u 
H > 

^ ►J 

u 

O <D C 
C O Q) 
O C 6 

§c 
U) 

C U 
O O W 
CC CO 
W Wi < 



65% 



80% 



95Z 





(Tl) 


(T2) 


(T3) 


65Z 


lA 


12 


11 




(TA) 


(T5) 


(T6) 


80% 


11 


10 


12 




(T7) 


(T8) 


(T9) 


95% 


lA 


12 


11 



The design establishes conditions in which students of varying ini- 
tial resource (prior achievement) combinations produce different levels 
(i.e. performance levels) of a uniform kind of output (matrix algebra 
skills). The range of observed resource/output combinations is undoubtedly 
greater than would ordinarily be observed in non-experimental conditions. 
The experiment is analogous to much agricultural research, or to ^'response 
surface'' methodology more generally, although a larger number of cells and 
observations would improve the analogy. 



Adminis t r a t ion 



In November, 1971, the experiment was conducted sequentially in the two 



c 
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schools, with the assistance of the regular classroom teachers and a gradu- 
ate student. Two weeks were required at each location. No alterations in 
school schedules were made for the experiment. Indeed, the somewhat compli- 
cated class schedules in both schools provided less than ideal experimental 

onditlons in terms of continuity and record keeping. All student-kept 
records were checked frequently for completeness and accuracy. On the whole 
the students cooperated extremely well and proved themselves excellent 
record-keepers. 



Summary 

A total effective sample of 110 eigth-graders was obtained in two 
schools rather less homogeneous in population than hoped for. Instruc- 
tional materials developed by James Block for a similar design but dif- 
ferent purpose were adapted for use here. Students were randomly assigned 
to one of three "mastery" or performance levels (defined as percentage 
correct on a twenty item test) for the first two, and again for a third 
lesson. Student self-paced instruction of the programmed- format lessons 
was administered by the experimenter and assistants, during separate two- 
week period^ at each school. 

The data available for analysis may be grouped into five categories: 

1) Student family background information as 
reported by student answers to questionnaire; 

2) Student SAT Arithmetic Concepts and Para- 
graph Meaning raw scores; 

3) Student performance level assignments; 



4) Student pretest, formative tests, and 
post-test scores; 

5) Student time (self-recorded) on all study 
and test activities. 

The analysis of these data is taken up in the following section. 



Ill 



RESULTS 



The first question to be taken up concerns the degree to which the 
ideal of the experimental design „as fulfilled. Then the results of 
several alternate multiple regressions with student time as the depen- 
dent variable will be presented and interpreted. Finally, certain sub- 
sidiary matters, such as the effect of performance level sequence on 
rate of learning, will be discussed. 

Student Mastery 

Students were randomly assigned to one of three performance levels 
for each lesson. But what was the actual or realized performance of the 
students? The experimental problem was to raise the performance of stu- 
dents whose initial formative test scores were below the assigned level, 
and to intercept students with satisfactory scores, so that they wouJd 
not exceed their assigned level. In operational terms, the ideal de- 
sired results were performance level scores with means of 13. 16. and 19 
respectively, and with zero variance. 

I was unable to obtain zero variance about the target scores, al- 
though in fact almost one-third (30.9%) of the students did attain ex- 
actly their pre-asslgned performnce levels on all three lessons. But 
I expect the reader to allow that zero variance, while ideal for theo- 
retical reasons, is perhaps too rigorous a standard to fulfill in prac- 
tice. I Offer in its place the concept of "minimal relative variance," 
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by which I .ean that the mastery procedure is effective when the variano. 
around sLudent target scores is snu.ll compared to the variance in the 
ability and prior achievement scores of the students. 

A comparison employing this minimal relative variance concept is 
made in Table 3. in „hlch-for each lesson and target performance level- 
the coefficient of variation of student final scores on each losso. is 
compared to that for student ability and prior achievement scores in each 



group, 



(INSERT TABLE 3) 



Table 3 



COMPARISON OF THE RELATIVE VARIATION IN 
STUDENT SAT, PRETEST, AND EXPERIMENTAL 
LESSON SCORES, FOR EACH LESSON AND PER- 
FORMANCE LEVEL ASSIGNMENT 



(2) (3) (4) 

Standard Coefficient of 

■ ^^'^^^^^^ Mean D eviation Variation (3-r2) 

PL13 ACONRW 18.135 5.779 0 319 

(65%) PRESOM 1.243 1.038 0*835 

Group TOTSCl 14.378 2.190 



PL19 ACONRW 19.103 6.711 



0.152 



0.329 



PL16 ACONRW 18.647 6 134 

(80%) PRESUM 1.265 1.163 0.919 

Group TOTSCl 16.000 1.181 0 074 



(95%) PRESUM "1:333 l^ll 

Group TOTSCl 18.308 1.575 0.086 



PL13 ACONRW 18.135 5.779 0 319 

(65%) PRESUM 1.243 1.038 o 835 

Group T0TSC2 13.595 2.254 0.*166 



PL16 ACONRW 18.758 6.165 0 329 

(80%) PRESUM 1.303 1.185 O.llO 

Group T0TSC2 15.849 1.278 0.081 



PL19 ACONRW 19. 

(95%) PRESUM 1. 

Group T0TSC2 18. 



270 6.854 0.356 

324 1.396 1.054 

595 0.832 0.045 



(CONTINUED) 



Table 3 (Continued) 
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2; 
o 
10 

w 



<1) (2) (3) (A) 

Standard Coefficient of 
^^^^^^^^ Mean Deviation Variation (3-r2) 



PU3 ACONRW 18.878 6.501 0.34A 

ieSA) PRESOM 1.293 1.123 0.869 

Group T0TSC3 14.293 1.692 0.118 



3 PU9 ACONRW 19.152 5.885 

(95%) PFiESUM 1.A55 i 325 

Group T0TSC3 18.758 0.*708 



g (80/^) PRESUM l.i7i 1.150 0.982 

Group T0TSC3 16.114 1.183 0.073 



0.307 
0.911 
0.038 



Definitions: PU3, etc. , is the assigned performance level; ACONRW is 
the SAT Arithmetic Concepts test raw score; PRESUM is the 
score on a jnatrix arithmetic pretest; TOTSCl, etc., is the 
actual final or "total score" attained on each lesson. 
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Note that in Table 3, Column 2, the mean "total score" (TOTSC) in 
each lesson should he, ideally, either 13, 16, or 19, according to the 
performance level assignment. In fact, the mean for the lower perfor- 
mance level tends to be nearer 14, that for the middle level is about 
on target, and that for the highest level tends to be about 18.5. Thus 
the procedures employed appear to be somewhat less effective at inter- 
cepting learning at the lower performance level than in raising it to 
the higher criterion levels. 

Column 4 gives the relative variation of the student scores in each 
group, for each lesson. Here we see that the coefficient of variation of 
scores on the experimental lessons is typically less than that of the stu- 
dents' SAT raw scores by a factor of 4 or 5, and less than the variation 
in protest scores by an even larger factor (except, again, for the lowest 
performance level) . 

I would ask the reader to mentally contrast such low-variance out- 
comes with conventional classroom procedure, in which typically a rela- 
tively wide-rather than narrow-range of outcomes is tolerated. Indeed, 
a wide range of outcomes is perhaps encouraged by some teachers, as an 
aid to "making a curve" for grading purposes. 

In summary, the instructional strategy employed for this experiment 
did not produce "ideal" results, but did in general obtain student per- 

narrowly concentrated near t arget scores , especially for the 
two higher performance levels. These student outcomes show far less vari- 
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ation than the pre-experimental measures of relevant student resources, 
and support the view that under appropriate conditions similar levels of 
mastery can be attained by students of widely varying initial abilities. 
Furthermore, these minimal variance outcomes near predetermined (i.e., 
exogenous) performance levels seem to reasonably approximate an economic 
model of production at various levels of a uniform kind of output. 

Selection of a Regression Model 

For the reasons outlined in my earlier paper, the simple linear ad- 
ditive regression model was rejected in favor of e cirvilJnear model, the 
so-called Cobb-Douglas function which, however, is linear in logarithms 
and so may conveniently be estimated by ordinary least squares methods. 

A single instructional technology was employed for the experiment; 
no comparison of technologies was attempted. Student "entry" or prior 
resource levels, as measured by SAT and pretest scores, were not manip- 
ulable for experimental purposes (apart from random assignment of stu- 
dents). Student target performance levels were manipulated, with consid- 
erable success as shown above. Thus the principal experimentally ob- 
served variable was student time spent at each stage. This situation 
is opposed to typical school practice, where student time spent on a 
topic tends to be bounded by the teacher's lesson plans or the pace of 



1 

One caveat to be noted is that the coefficient of variation may be 
biased if a test Is subject to a "ceiling effect." This problem and 
other assessments of the effectiveness of the mastery learning proced- 
ures employed here are discussed in the detailed report of this re- 
search (see note 1, page 1). 
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the class as a whole, and student performance is a variable. Here, stu- 
dent achievement was exogenously determinec:. .n H .tudent time u^^ ^^w^ 
to_varx. Therefore, the basic function to be estimated was: 

Y - a + bjXi + fcjXj + u (10)*' 
where: Y = log^time in minutes 
a = log^constant term 

^i " logs^of a vector of student test scores 

Xj = logs^of a vector of treatment 

(student assignment) dummies 

u « log residual, 
e 

Transformations of Selected Variables 

Student SAT Arithmetic Concepts and Paragraph Meaning r£«w scores were 
converted into ratios according to the formula r = (p / i - p) , „here p 
U the proportion the raw score bears to the total points possible on the 
test. The logarithm of r was then taken. Thir monotonic tr^msformation, 
known as the "logit" transformation, was used to stretcl) out the upper end 
of the distribution and obtain a better conditioned matrix for regression 
procedures. The logit values thus obtained were used directly in the re- 
gressions. 

The student pretest variable (PRESUM) was also transformed slightly, 
according to the formula: NEWPRE - (PRESUM + 1). This was necessitated 
by the presence of a pretest score of zero for some students, since the 
logarithm of zero is undefined. The natural logarithms of resulting 
NEWPRE scores were used in regressions. 
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Dunroy Variables 

A number o£ "dunnay variables" were constructed In order to include 
certain qualitative states in the regression fmalysis.^ In general, N 
"stales" or "qualities" may be represented by (N - 1) dummy variables, 
coded ] or 0 to represent the presence or absence of the quality. For 
example, this experiment involves nine treatment groups. (Recall that 
students were assigned to one of three target performance levels for a 
pair of lessons, then were assigned to one of three levels for a third 
lesson. Thus a 3 x 2 assignment matrix results, containing 9 (9 = 3 x 3J 
cells.) pplying the rule for construction of dummy variables, these 
nine cells can be represented by eight (8-9-1) dummies, coded 1 if 
the student is a member of the cell, 0 otherwise. In this fashion, the 
students of all but one cell are identified by a "i" signifying member- 
ship in a particular cell. Students in the remaining cell receive a "0" 
for all eight of th-2 dummy variables, signifying membership in none of the 
other eight cells. This cell becomes the reference group against which 
membership in the other cells is measured. Thus the regression coef- 
ficient for each cell or treatment dummy represents the net effect of 
being in that cell compared to the reference group. The particular cell 
to be used as the reference group is an arbitrary decision, and may be 



1 

An explanation of the use of duamy variables may be found in Jacob Cohen, 
Multiple Regression as a General Data-Analytic System," Psychological 
BuUetin. Vol. 70, No. 6 (1968), A26-A3, reprinted in Bernhardt Lleberman, 
Contempora ry Problems in Statisti cs (New York: Oxford University Press 
1971), A21-AA0. ' — 
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selected for analytical convenience. 

The use of dunany variables in this way can be shown to be equivalent 
to analysis of variance procedures, but the multiple regression context 
has the added advantage of including quantitative variables as well.' 

R egressi on Results 

We have at last surmounted the preliminary ubstaclcs (I hope to the 
reader's satisfaction), and can now proceed to identify a production func- 
tion as promised in the title of this essay. Wo .<-h.u take a direct route 
to the more interesting results.^ 

We naturally expect students with higher leve.s of relevant prior 
achievement to spend less time on new learning, other things equal. This 
expectation is borne out in Regression ^1. as shown by the negative ..lope 
coefMcients for variables three, four. ,nd five. Thus, as the .student's 
Arithmetic Concepts. Paragraph Meaning, or pretest score is higher, the 
student will spend less time for a given an^^unt of new learning, and con- 
versly. it should be remembered that the regression equation coefficients 
are for logarithms of .the variables; interpretation will be discussed below. 

Variables 6 through 13 are treatment dummy variables, corresponding 
to the treatment matrix cells shown in Table 2 (see page 13). a.s Tl. 12. etc. 



1 

Ibid, 



2 

AddiU„„.l ,es„Us, testa, and r.H^^nt. vUl be Included 1„ the detailed 
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Treatment f , the 80% performance level assignment for all three lessons, 
was selected as the reference group in construction of dummy variables. 
As sliown in Regression #1, the t-statistics for tlie individual treatment 
variables are generally not high, with the exception of that for T9, which 
is significant at the .01 level. Taken as a group, however, the treatment 
dummy variables do contribute significantly to the total variance explained. 
This is shown in the following manner: Regression //2 is similar to Re- 
gression //I, below, but omits the treatment dummy variables. As the treat- 
ment assignments are independent of the variables in Regression //2, an 
statistic may be computed to estimate the significance of the additional 
variance explained by the addition of treatment variables in Regression , 
as compared to Regression //2. The additional variance explained is almost 
10%, and the F-statistic for this increment is 2.49 (8,95 d.f.), which is 
significant at p-^^.OS. 
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Further explanation of the variance in total time spent by students 
is obtained by adding dummy variables to represent school and sex effects- 
Sex differences in school learning behavior are familiar, with the general 
finding Lhat at least through grade school (including eighth grade) girls 
tend to receive higher marks than boys, to be disciplined less often, and 
so on. Thus to capture any systematic differences associated with sex, a 
single dummy variable (SEX) was coded 1 for girls and 0 for boys. If, as 
migia be expected, girls use less study and test time than boys, the re- I 
suiting regression coefficient will have a negative sign, 

A dichotonous variable (SCHL) was created to capture any effects as- 
sociated with the two schools. Presumably any systematic differences be- 
tween the schools (if uncorrelated with other variables in the study) will 
be captured by this variable, although it might be impossible to specify 
exactly the source of the difference within the school settings. Tlius 
such a variable is not analogous to an experimental "treatment," as no 
specific treatment components are identified. Instead, it is analogous 
to a machine or plant dummy in a regression analysis of production: a 
host of unspecified differences may be grouped by some readily identifiable 
feature, in this case by school. 

Regression //3 is similar to Regression #1, but adds the school and 
sex dummy variables. The R^ has now risen to 0,62, an increase of ,05. 

i 

This gain has a significant F-statistic of 6.25 (2,93 d.f.). The other 
variables in the regression retain relatively their same coefficients and 
t-values. SEX has a statistically significant slope coefficient of nearly 
the same magnitude as the Paragraph Meaning coefficient. Its negative sign 

/ indicates that girls tend to take less time to criterion than boys, other 

things equal. 

O , (INSERT REGRESSION #3) 
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The SCHL variable coefficient in this regression does not reach con- 
ventional levels of statistical significance, but is large enough to sug- 
gest that systematic differences between the groups may be correlated with 
cheir school of attendance. (In other regressions to be reported, SCHL 
does enter with statistical significance.) There are a number of possible 
such differences. Different average mathematical and reading achievement, 
as indicated by SAT scores, is an obvious candidate. These scores in turn 
may be associated with different family incomes and preferences in the two 
school attendance areas-differences which affect student access to learn- 
ing resources at home and motivation and performance in school. 

Another difference between the groups may be a "teacher effect." All 
students in the experiment had mathematics with their same teacher the pre- 
vious school year: each school has only one teacher for seventh and eighth 
grade mathematics. Thus teacher influence on student work habits or study 
skills (or on unmeasured mathematical knowledge) may account for "school" 
differences. 

A final source to be mentioned of possible difference between schools 
is that "the" experiment itself may actually be two experiments. The 
schools were visited sequentially, and some changes in the experimenter's 
behavior, "learning" how to administer the experiment, if you will, may 
have influenced outcomes. Every attempt was made to make administration 
uniform between schools, but discrepancies, no do ubt^ crept in. 

A large number of other variables were available for inclusion in re- 
gression analyses. These include several family environment variables, 
coded for the most part as duimny variables. These were: DESK, indi- 
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eating presence or absence of a desk or table the student considers his 
own at home (a home resource for school-related behavior); FATHER, MOTHER, 
indicating presence or absence of each parent (natural or by adoption); 
READTO, whether or not the child was read to daily while of pre-school age 
.according to his recollection; MOTHASP, FATHASP, indicating the student's 
perception of whether or not the parent expects him to enter college; and 
TALK, wh(ither or not student and parents frequently discuss his schoolwork 
Also included were the logs of two numerical variables: LNFED, log of 
father's education ir. years; and LNEDSP, the log of student's expected 
schooling in years. 

It was not anticipated that these variables, or many of them, would 
have strong impact in terms of regression analysis. Although a plausible 
argumenl can be made that most of them should have some bearing on stu- 
dent achievement or time to criterion, such influence might not be clearly 
revealed in an experiment of relatively short duration which departs from 
conventional school procedure and in which homework as such plays no part. 
Tn addition, much of this information was collected in the hope that the 
original project design (a sample including strongly contrasting socio- 
economic groups) would eventually be fulfilled. In the absence of a con- 
trasting group, most of these variables show relatively little variance, 
and hence cannot have much explanatory power in the present regression. 

The results of a regression including the earlier variables and 
those just discussed are presented in Regression //4. Among the new vari- 
ables added, only the log of father's education in years approaches sta- 
tistical significance. (Mother's education is highly correlated with 
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father's education, but is less statistically significant than father's 
education when substituted in the below and other regressions.) In- 
clusion of the eleven new variables does increase the slightly over 
that of Regression if 3, but the F-statistic for the increase is only 0.64 
(11,82 d.f.). Thus no Importance can be attached to the small gain In 
variance explained. 



(INSERT REGRESSION U.) 
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Summing up, we have seen that more than 60% of the variance in time 
spent by individual students can be explained with knowledge of achieve- 
ment test scores, sex, school, and performance level assignments* 

The regressions reported above show, for a given technology, the rela- 
tive influence of prescribed performance levels and varying student re- 
sources on tota] study and testing time spent by students. Because they 
embody this information, they may be interpreted as educational production 
functions. It might be more accurate to call them ''time functions," since 
the dependent variable is elapsed time which, strictly speaking, is allo- 
cated or ''spent" by students rather than "produced." The estimated equa- 
tions probably should not be called "cost functions" because they include 
no factor prices, for the simple reason that none exist for such variables 
as student achievement test scores. Nevertheless, there is a fundamental 
duality between cost and production functions, and with knowledge of prices 
or appropriate assumptions about them, cost functions can readily be ob- 
tained from production functions."^ As the equations estimated here reflect 
underlying technical production relationships in the setting described, they 
will be referred to "production functions." In particular, I nominate Re- 
gression If 2 as "the" production function identified in this paper. 

Intert ::etation of Regression Results 

The use of elapsed time as the dependent variable here facilitates a 



1 

R.W. Shephard, Cost and Production Functions , Princeton: Princeton Univer- 
sity Press, 1953. 
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rather straightforward interpretation of the regression results. For dis- 
cussion purposes, we select Regression #3 as containing the most readily 
interpretable and generally most significant variables. Then the pre- 
dicted (log) time to completion for a student with an^ combination of 
charactoristics (within the range observed in this study) is found by in- 
serting the appropriate student scores into the equation estimated by 
Regression //I. 

Suppose, for example, we want to estimate the required time for a 
"high-ability" student, Haynes, to complete the learning sequence with 
an 80% (middle level) performance requirement throughout. To be specific, 
it is assumed that Haynes' SAT raw scoifes are one standard deviation above 
the sample mean, he is male, attends the first of the two schools studied, 
and scored two correct answers on the matrix arithmetic pretest. The fol- 
lowing are the values to be inserted in the equation: 

1. LNPRE = 1.098 (Natural log of pretest score transformed by 

adding 1; i.e. in this case log 3) 

2. LGTACN = 0.51 (Logit of Arithmetic Concepts raw score of 

25; approx. equal to mean plus one s.d.) 

3. LGTPAR = 0.93 (Logit of Paragraph Meaning raw score of 

43; approx. equal to mean plus one s.d.) 

4. SEX = 0 (Males are coded zero.) 

5. SCHL =1.0 (The first school studied is coded 1.) 

6. T^ = 0 (The reference group assignment was chosen 
through here, hence all treatment dummies arc coded zero. 

These scores are multiplied by their respective (estimated) coeffi- 
cients, and added together with the constant term to give the estimated 



Log time for this student. The resulting term is exponentiated to give 
time in minutes. These operations are carried out In Table A, and give 
.)s ostinuired time of lOA minutes for Haynes. 



(INSERT TABLE A) 
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For comparison, imagine now a student similarly situated but whose 
prior achievement scores are one standard deviation below the sample means 
dnd whose matrix algebra pretest score is zero, ilow much longer will it 
take this "low-ability" student, Lester, to attain the same criterion per- 
formance levels as Haynes? The answer is found by substituting new values 
where appropriate, as in Table A. We obtain an estimated time of ?0A.4 
minutes for Lester. Thus equal final attainment can be had fo r both low 
and high ability students, but to do so the low student needs almos t twjcy 
the time spent by the high student . If the cost of time in ihi.s techuolo^ 
were constant, then it would be almost twice as expensive to obtain the 
given level of performance from the low ability student as from the higlj. 
It will be shown, however, that in all probability the additional tl*Ti<. i 
quired by the low student is also more costly per unit. 
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In the examples of Tables A and 5, the sex and school effects would 
be as follows: A high-scoring girl would take about 14 minutes fewer than 
the high-ability boy; the low-scoring girl would finish about 27 minute.s 
sooner than her male counterpart. That is, the girls wuld take about 
13% less time than the boys to reach U same performance level. Both the 
high and low-scoring boys would take about 9% less time in the second school. 

uaiat is the effect of performance level assignment on total time r.c, 
completion? To answer this question for a student of given characteristics, 
we need only repeat the procedure used in Tables A and 5, substituting new 
values wh-re required, including the appropriate non-zero treatment dummy. 
This time an "average" male student will be assumed: one whose pretest, 
arithmetic concepts and paragraph meaning scores are each at the sample 
mean. If the treatment dummies all had statistically significant slope 
coefficients, any such coefficient could be used to represent the effect 
on required time of the corresponding performance level sequence. .Since 
few of the dummies have consistently significant coefficients, the example 
here will compare only Treatments 1, 5, and 9, in which students were as- 
signed to the low, middle, or high performance levels throughout the entire 
sequence. Thus the values to be substituted in the estimated equation are 
the mean values of the LNPRE, LGTACN, and LGTPAR variables (0.6886, -0.1370. 
0.2364, respectively) and the appropriate dummy variable coefficients. 

Tlie resulting estimated times for the "average" male student in the 
first school are: 

Tl (65% performance level): 137 minutes 
T5 (80% performance level): 143 minutes 
T9 (95% performance level): 199 minutes 
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Thus there is little difference in impact on time between the low and 
middle performance level sequences, but the time required in the high 
level sequence is about 40% greater than for the middle sequence. 

The time spent by above and below-average-ability male students in 
the middle performance level sequence was presented in Tables 4 and 5. 
Predicted times for such students in the low and high performance se- 
quences have also been calculated, and are combined in Table 6 with the 
below results for an average student. 



(INSERT TABLE 5) 
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Table 6 

PREDICTED TIMES IN THREE CRITERION 
PERFORMANCE LEVEL SEQUENCES BY 
LOW, AVERAGE, AND 
HIGH ABILITY 
STUDENTS 



(Tl) 

LOW: 195m. 
AVG: i37iu. 
HIGH: 99m. 




;(T3) 


(T4) 


(T5) 

LOW: 204m. 
AVG: 143in. 
HIGH: 104iii. 


(T6) 

1 


(T7) T 
■ L 


(T8) 


(T9) 

LOW: 284m. 
AVG: 199m. 
HIGH: 144m. 
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PerlMps the ^ost striking finding to emerge from Table 6 is that a 
hl^hz^hJJ^t^LM^^ complete the thr..-lesson_se^nce at the hi.h..^ 

com£letejhe_seau^ The 'Wrage" 

student cakes only slightly longer than the low-scoring pupil in the same 
comparison.. Thus the table dramatically illustrates the importance of al- 
location Of time for the achievement of specific skills by persons of differ- 
ent ability, in this context, the allocation of time i.s equivalent to the 
allocation of school and student resources. In particular, schools seeking 
uniform criterion performance among students (of the kind studied here) m 
matrix arithmetic, using the technology employed here, would have to allo- 
cate .space, time with materials, and "teacher" time to low-ability stadents 
at about twice the rate for high-ability students. 

furt her Regression Results : 

Interactions. One purpose of che 3 x 3 factorial design of this ex- 
periment was to allow statistical testing of whether the sequence of per- 
formance level assignments has a significant effect on time to criterion, 
apart from the magnitude of the performance levels themselves, in the 
language of analysis of variance, such sequence effects would be called 
iaLiEictions. No significant interactions were found. 

The procedure for investigation of interactions was not conventional 
ANOVA. but rather an exactly equivalent procedure by means of linear re- 
gression. In the regression approach to analysis of variance. dunm>y vari- 
ables represent main effects and interactions, and significance may be 
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tested by means of the F-test for addition to explained variance. The 
steps to be followed are these: 

1. Regress covariates alone; 

2. Regress covariates and dummy variables for main effects; 

3. Regress covariates and duirany variables for treatment cells. 

2 

Then an F-test for addition to R from Step 1 to Step 2 tests the signi- 
ficance of main effects. Likewise, the increase from Step 2 to Step 3 
is tested for significance of interactions. These results are summarized 
in Table 7. 

Table 7 

TESTS FOR SIGNIFICANCE OF MAIN 
EFFECTS AND INTERACTIONS 



Regression 


r2 


2 

F-test for significance of addition to R 


1. Covariates only 


0.5734 


27.15 (5,101) p<.005 


2. Covariates plus 
main effects 


0.6279 


3.52 (4,96) p .01 


3. Covariates plus 
all treatment 
cells 


0.6467 


1.22 (4,92) p>.05 



What does lack of significant interaction mean? It does not mean that 
no interactions whatsoever are present* It does mean that the dam reveal 
no interactions at a conventional level of significance, in this case the 
•05 level. It is possible that observation over a longer sequence of in- 



44. 

srructional units, or for different criterion performance levels, would 
reveal effects pertaining to performance level sequence. An example of 
such a sequence effect, or interaction, would be if students assigned to 
the highest performance level for the first two units of instruction took 
significantly less (or more) time to criterion in the third lesson than 
did students with lower prior performance level assignments. In the pre- 
sent experiment, however, such effects were not detected. 
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DISCUSSION 



In this experiment, about two-thirds of the variation in total study 
and testing time spent by students was explained by student characteristic 
and performance level assignment variables. The estimated production func- 
tion was used to project the time needed by students of various initiaJ 
resource combinations to achieve prescribed levels of mastery. 

If the cost of time were uniform, it would be twice as expensive to 
obtain a given level of mastery by a below-average student as by an above- 
average student.^ In fact, the cost of time was not uniform, as the lower- 
ability studfiLs required more frequent and prolonged assistance with vari- 
ous aspects of the instructional material than did the superior students. 
Thus tlie slower youngsters were also more expensive per minute to teach. 

Most studies conclude that yet more research is needed. I will not 
let the triteness of such a conclusion deter me from asserting it here. 
The findings of this study strongly suggest that under appropriate condi- 
tions the allocation of student (and teacher) time can be a school policy 
variable of decisive Importance. This study suggests that the goal ot 
equity in achievement in specific subject areas or skills will require 
massive reallocations of school resources. Suppose, for example, a "right 
to read" program were to be rigorously undertaken. What would an effec- 



1 



Here, above- and below-averages refer to SAT and pretest scores one stan- 
dard deviation above and one standard deviation below the mean of students 
m this study- 
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tivo program cost? The answer, 1 think, would lie in the technology to 
be U4>ed, the distribution of client entry resources (prior achievement, 
etc.)> and in the minimum performance level adopted. It is quite pos- 
sible, of course, that the minimum performance levd adopted would it- 
self depend on the relative cost of achieving various performance levels. 
Thus, if there was not a willingness to "pay any price" for literacy, the 
minimal level chosen would be such as to balance feasible literacy rates 
against what the public was willing to pay for improved literacy. 

The data presented here are limited in several ways: only one t ch- 
nology and one subject matter were used; affective issues were ignored; 
the experiment was of relatively short duration; the students represent a 
narrow range of home backgrounds. I suggest, however, that many of these 
"limitations" are in fact strengths because they are known and because they 
lend precision to the findings. Nevertheless, many additional issues must 
be explored before the usefulness of this approach can be determined. The 
inclusion of students of contrasting socio-economic backgrounds should be 
a matter of high priority on a research agenda. Likewise, the effect of 
different technologies and subject matters on the parameters found here 
should be investigated. We certainly want to know what happens over great- 
er periods of time: Does the bright child's advantage in rate of learn- 
ing persist, so that the academically rich grow richer, while the poor grow 
(relatively) poorer? Perhaps the bright youngster's learning is asyraptotic 
in specific situations, so that his advantage lies primarily in the great- 
er variety of subjects he can master while the below-average child is work- 
ing to master just one. Finally, other age groups must be studied. These 
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and other matters for investigation should yield findings of use in the 
analysis and planning of educational policy. 
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